The present disclosure relates to an image processing apparatus and image processing method and program, and more specifically, it relates to an image processing apparatus and image processing method and program wherein pixel value reading of a shot image is executed as sequential readout processing.
As related art, an overview of two techniques below of exposure control processing will be described in the order of (1) Focal-plane shutter operation and occurrence of distortion, and (2) Exposure time control (shutter control) to set exposure times by different regions.
(1) Focal-Plane Shutter Operation and Occurrence of Distortion
First, focal-plane shutter operation and the occurrence of distortion will be described. A shutter operation that controls exposure starting and exposure ending from one direction of an imaging device face is a shutter operation method of image shooting processing with an imaging apparatus. This shutter operation is called a focal-plane shutter operation or a rolling shutter operation. A feature is that if exposure starting and exposure ending is controlled from an upper row of the imaging device towards a lower row, for example, exposure time between rows shifts slightly.
A CMOS image sensor configuration and shooting processing example will be described as an example of an imaging device having a focal-plane shutter operation function, with reference to FIG. 1. FIG. 1 is a diagram showing a partial configuration of an imaging device (CMOS image sensor) 101. The imaging device (CMOS image sensor) 101 is configured to have a vertical scanning circuit 102, horizontal scanning circuit 103, and multiple pixels 104 that have been disposed in an array.
Within the pixels 104, a charge is accumulated in a photodiode by an exposure processing that accompanies the shooting of a subject. The charge accumulated in the photodiode of each pixel is output to a vertical signal line 113 via an amplifying transistor and transfer transistor. The signal current output to the vertical signal line 113 is further supplied to the horizontal scanning circuit 103, and upon a predetermined signal processing having been executed, is output externally via a signal output line 114.
The pixels arrayed vertically are connected in common to the vertical signal line 113, so in order to independently read out the signals of each pixel, only the signal for one pixel should be output at one time to the vertical signal line 113. That is to say, with the imaging device (CMOS image sensor) 101, as shown in FIG. 2A, for example, a signal is first read out from each of the pixels 104d arrayed in the lowest row, and next, readout is performed from the row of pixels 104c as shown in FIG. 2B, and subsequently the readout rows are changed and signal readout is performed, thereby enabling independent readout of signals of each pixel. The control signals for the pixel readout are output from a horizontal reset line 111 and horizontal selection line 112 connected to the vertical scanning circuit 102 shown in FIG. 1, for example.
The pixels 104 making up the imaging device (CMOS image sensor) each start exposure again, immediately following the readout processing of the accumulated charge. That is to say, exposure processing for the next image frame is started.
Thus, upon the readout processing being executed sequentially by row, and immediately thereafter exposure processing being started, differences in the starting point-of-time and ending point-of-time of exposure, i.e., a shift in exposure time (or exposure period) occurs between the photodiode 104a of the head row and the photodiode 104d of the last row. This is a feature of a shutter operation called a focal-plane shutter operation or rolling shutter operation.
Note that the diagram only shows the four rows of 104a through 104d, but this only shows a portion of the imaging device, and an actual imaging device has a great number of rows set, such as several hundred to several thousand rows, and sequential readout is executed by each row.
An example of starting and ending timings of exposure of each row and the charge readout starting timing will be described with reference to FIGS. 3 and 4. FIGS. 3 and 4 both show the temporal axis on the horizontal axis and the row on the vertical axis.
For example in FIG. 3, the charge readout timing has a timing shift occurring by row, as shown with the dotted line 151a, 151b shown in the diagram. Multiple rectangular blocks shown in FIG. 3 show the exposure time of a certain shot image frame, and are the exposure times by row block made up of a row or multiple rows.
Exposure processing is started immediately following the timing shown in the readout line 151a of the first shot image frame. As shown in the readout line 151a, the exposure start time becomes time that is slightly shifted for each row. In the graph shown in the diagram, the row on the upper side first has exposure started, and the lower the rows are, the later the exposure starts. The uppermost row has an exposure starting time of time (t1), and the bottommost row has an exposure starting time of time (t2).
The right edges of the multiple rectangular blocks shown in FIG. 3 are the timings of the readout processing of the exposure image to be executed, and accumulated charge of the pixels for the rows is read at the timings shown by the readout line 151b. 
In this example, the exposure ending time is approximately the readout processing time, and as shown by the readout line 151b in FIG. 3, readout processing for each pixel is performed by row, sequentially from the head row. On the uppermost row, time (t2) is the exposure ending time, and on the bottommost row, time (t3) is the exposure ending time. Note that with this example, the exposure starting and exposure ending for each row has the same timing shift for each row, so the exposure time for all of the rows is the same.
FIG. 4 shows the exposure processing and readout timing corresponding to two continuously shot frame images at the time of motion shooting. As shown in FIG. 4, the period of the readout line 152a through readout line 152b is the exposure time for the head shooting frame N, and the pixel value readout is executed from each row at the timing shown by readout line 152b. 
The period of the readout line 152b through readout line 152c is the exposure time for a trailing shot frame N+1, pixel value readout is executed for each row at the timing shown by readout line 152c. 
In the example shown in FIG. 4, for the head shot frame N, the exposure starting time is time (t1a) for the uppermost row and time (t1b) for the bottommost row, and the exposure ending time is time (t1b) for the uppermost row and time (t1c) for the bottommost row. For the trailing shot frame N+1, the exposure starting time is time (t2a) for the uppermost row and time (t2b) for the bottommost row, and the exposure ending time is time (t2b) for the uppermost row and time (t2c) for the bottommost row.
In the example shown in FIG. 4, for example the exposure time of the bottommost row of the head shot frame N and the exposure time of the uppermost row of the trailing shot frame N+1 are roughly in the same timeframe. That is to say, a phenomenon occurs wherein the image data on the lower side of the head image frame and the image data on the upper side of the trailing frame are images in roughly the same timeframe.
As a result, for example, in the case of imaging a subject having movement, or in the case of performing shooting processing such as moving the camera itself during exposure and shooting, distortion occurs in the image from shifts in the exposure time between rows resulting from the focal-plane shutter operations.
An example of image distortion will be described with reference to FIGS. 5A through 5D. FIG. 5A is a photograph example in the case of shooting with the camera in a stopped state. FIG. 5B is a photograph example in the case of shooting while moving the camera in a horizontal direction. The image in FIG. 5A has no distortion occurring, but the image shown in FIG. 5B has distortion occurring.
Similarly, FIG. 5C is a photograph example in the case of shooting while a car is in a stopped state. FIG. 5D is a photograph example in the case of shooting while a car is in a moving state. The image in FIG. 5C has no distortion occurring, but the image shown in FIG. 5D has distortion occurring.
Such distortion occurs due to shifts in exposure time of the imaging devices described with reference to FIGS. 3 and 4, i.e., due to exposure time differing a little at a time from the upper edge row to the lower edge row. The distortion occurrence phenomenon is called a focal-plane shutter phenomenon or a rolling shutter phenomenon.
Related art for reducing such distortion by a focal-plane operation will be described. For example, Japanese Unexamined Patent Application Publication No. 2004-140479 discloses a method of reducing distortion of a subject having movement, in which reset operations and readout operations by imaging devices are performed at high speed, the image data read out at high speed is temporarily stored in a storage device, and the stored data is read out at a slower frame rate and output downstream.
The method described in Japanese Unexamined Patent Application Publication No. 2004-140479 has to have high speed readout operations performing in order to reduce distortion. However, high speed operations are restricted, so completely eliminating distortion is impractical. Further, a secondary problem occurs, which is that power consumption increase and noise increase occurs due to the high speed operations.
Also, Japanese Unexamined Patent Application Publication No. 2004-140149 discloses a technique for adding transistors used for global shutter operations, inside pixels. However, the disclosed technique in Japanese Unexamined Patent Application Publication No. 2004-140149 has to have a transistor added so the pixel size of the imaging device increases, and is restricted by not being applicable to use in a small image sensor or a mega-pixel image sensor.
Also, Japanese Unexamined Patent Application Publication No. 2006-148496 discloses a configuration to reduce distortion by taking in an output signal from an image sensor to a storage apparatus, and generating one image from multiple frames.
The method in Japanese Unexamined Patent Application Publication No. 2006-148496 will be described using FIG. 6. FIG. 6 shows exposure time by row for three consecutive shot frames at the time of motion shooting, the three frames being frame N−1, frame N, and frame N+1. As shown in FIG. 6, with an imaging device (CMOS image sensor), shooting is executed with a focal-plane shutter operation, and exposure timing differs by row. Accordingly, distortion such as that described earlier with reference to FIGS. 5A through 5D, i.e. image distortion resulting from movement of the object or the camera itself occurs. Thus, the images shot at before and after times are used, interpolation is performed which takes time into consideration, and an image is generated and output which is equivalent to that wherein an image of all of the rows of one image frame has been shot at the same time at a certain time.
For example, in the case that the three images of frames N−1 through N+1 are shot with the settings in FIG. 6, correction of the image in frame N is performed, and a corrected image similar to that shot at the same time as a timing T0, which is in the center position of the shooting time, is generated. In this event, correction processing is performed with reference to the image in frame N−1 and the image in frame N+1.
The technique described in Japanese Unexamined Patent Application Publication No. 2006-148496 has the advantage that computation is simple, since an image is generated by linear interpolation between frames. However, a storage apparatus (memory) serving as a frame buffer has to be provided. Also, the processing is not to eliminate distortion, but to cause the distortion to be unnoticeable by blurring the distortion, so there is a problem in that the screen blurs greatly if an object or the camera moves greatly.
For example, in this case of an image shot with the settings in FIG. 6, in the generating processing for the corrected image of the frame N, the row on the upper edge of the image is created by interpolation using weighting of approximately the same amount for each of the image of the frame N and the image of the frame N+1, and the row on the lower edge of the image is also created by interpolation using weighting of approximately the same amount for each of the image of the frame N and the image of the frame N+1. By performing such processing, the amount of blurring due to movement of objects increases at the upper edge and lower edge of the screen. However, the center portion of the screen is approximately interpolated by the image in frame N, so the center of the screen only blurs as before, so there is a problem in that the amount of blurring greatly differs depending on the position on the screen.
Also, Japanese Patent Application No. 2007-208580 discloses a configuration to reduce distortion wherein the output signal of the imaging device is temporarily taken into a memory, motion vectors are detected for each divided region of multiple consecutive photograph images stored in the memory, and one corrected image is generated while performing position correction.
With the method in Japanese Patent Application No. 2007-208580, blurring does not occur from correction processing as with the above-described Japanese Unexamined Patent Application Publication No. 2006-148496, but there are problems in that computation of detecting the motion vectors is complicated, and in the case that computing the motion vector fails, visually perceivable image breakdown occurs.
Further, Japanese Unexamined Patent Application Publication No. 2007-336314 discloses a configuration to reduce distortion with focal-plane operations, wherein a great number of images taken consecutively are taken in to the memory by high speed operations of the imaging device, for example, and one corrected image is generated from these images.
The method in Japanese Unexamined Patent Application Publication No. 2007-336314 is a configuration to generate an image with linear interpolation, similar to the method in Japanese Unexamined Patent Application Publication No. 2006-148496, but has the advantage in that the amount of blurring in the entire screen due to high speed operation of the imaging device is negligible, and distortion can be corrected well.
However, with the method in Japanese Unexamined Patent Application Publication No. 2007-336314, high speed operation of the imaging device is a premise, so similar to the configuration in the above-described Japanese Unexamined Patent Application Publication No. 2004-140479, power consumption increase and noise increase become problems.
(2) Exposure Time Control (Shutter Control) to Set Exposure Times by Different Regions
Next, exposure time control (shutter control) to set exposure times by different regions will be described. The exposure time as to each pixel of the imaging device can be controlled to expand a dynamic range of the shot image.
In a bright subject region, when the exposure time is long, the accumulated charge of the pixel saturates, and an accurate pixel value is not obtained. On the other hand, in a dark subject region, a longer exposure time enables a more accurate pixel value to be obtained, corresponding to the subject brightness. Thus, in a region where the subject is bright, a pixel value of a pixel set for a short exposure time is obtained as a valid pixel value. On the other hand, in a region where the subject is dark, the pixel value of the pixel having a long exposure time is obtained as a valid pixel value. These are combined to generate an output image. Note that at the time of output of a final pixel value, pixel value adjustment processing based on the exposure times is executed.
Japanese Unexamined Patent Application Publication Nos. 2006-253876 and 2006-542337, and Japanese Patent Application No. 2008-147818 disclose techniques to expand the dynamic range of a shot image, setting the exposure times that differ by region of the imaging device. For example, the configuration sets a short time exposure row and a long time exposure row in every other row of the pixel rows of the imaging device.
For example, Japanese Unexamined Patent Application Publication No. 2006-253876 discloses a configuration wherein electronic shutter operations of a CMOS image sensor are operated with the even-numbered rows and odd-numbered rows operating alternately, thereby setting high-sensitivity pixels (long time exposure pixels) and low sensitivity pixels (short time exposure pixels), and enabling imaging of a high dynamic range image by combining pixel values according to the subject brightness.
Japanese Patent Application No. 2008-147818 discloses a configuration wherein, in addition to the configuration in Japanese Unexamined Patent Application Publication No. 2006-253876, modification by row is also further enabled for readout timing.
Japanese Unexamined Patent Application Publication No. 2006-542337 discloses a configuration wherein, in an imaging device having a color filter with a Bayer array, two patterns of exposure time are set for each row, or for multiple rows of more than one, and changes are made with electronic shutter operations.
The configurations in Japanese Unexamined Patent Application Publication Nos. 2006-253876 and 2006-542337, and Japanese Patent Application No. 2008-147818 execute exposure time control by region, with a configuration using an electronic shutter.
(3) Overview of Related Art
As described above, with a configuration using a focal-plane shutter, shifting in exposure periods by row occurs, for example, and the fundamental problem of distortion occurring that results from this shifting is not resolved. Also, techniques to perform exposure period control by region and expand the dynamic range are used, but with these configurations also, the shifting of exposure period by row is not prevented in the case that a focal-plane shutter is used, and the problem where distortion resulting from this shifting occurs is not resolved.